Unifying Low-Level Vision

This white paper supports the goal of establishing Computer Vision as a coherent intellectual discipline by suggesting a specific agenda for the unification of many low-level vision principles, algorithms, and data structures. Our goal is to identify a set of highly related low-level vision problems, define their common structure, and establish a coherent intellectual discipline around the shared structures. We believe this can unify and simplify the understanding, teaching, and quality of low-level computer vision as a science. There is a set of low-level vision tasks that are closely related and share a small set of common principles and yet, are developed mostly separately in the literature. These tasks are background subtraction, tracking (both near-field and far-field), stereo vision, image stitching, image registration (both medical and nonmedical), object recognition of “low variance” object classes (such as rigid objects), optical flow, and general alignment algorithms. The literature on background subtraction, for example, has fewer references to tracking than one would expect given their closely related nature. At the heart of these tasks is a low-level image representation and an image comparison function giving some notion of similarity or distance between two images or image patches. There has been tremendous innovation in developing low-level image representations and image comparison functions. Some examples include SIFT descriptors [6], HoG descriptors [3], geometric blur [1], the pyramid match kernel [5], mixtures of Gaussians at each pixel [8], non-parametric distributions at each pixel [4], affine invariant descriptors [7], and many others. We would like to revisit these descriptors and ask the question, “What properties do we want such a descriptor, and its associated comparison function, to have?” Developing primitive functions by specifying their requirements first has a history of success in the mathematical sciences. Two salient examples include Shannon’s definition of information entropy and Einstein’s development of Special and then General Relativity. Shannon required that a measure of information be continuous, symmetric, additive, and should be highest when all outcomes are equally likely. These requirements constrain the definition of entropy up to a positive constant (the units). Similarly, Relativity essentially follows from the simple Principle of Equivalence of reference frames. In computer vision, this approach is highlighted by such foundational works as Canny’s edge detector [2], defining the optimal edge detector with respect to certain requirements. In a similar spirit, we would like to return to the requirements of low-level descriptors and comparison functions in view of developments from the last 20 years. What should the requirements of low-level representations and comparison functions be? Some candidates include • robustness to image noise, • robustness to small misalignments, • a multi-scale or multi-resolution aspect, • absence of “hard bins”, which introduce oversensitivity to position, • “denseness”, the property that every location in the image contributes information to the descriptor,